Safety - Critical Systems : Prescient , Presignifying , Public - Good Systems

The emergence of safety-critical systems and their much desired institutionalisation involves domainand applications-independent issues that, in some sense, can be related to the interdependent issues of knowledge, language, and ethics. The design, maintenance, upgrading and decommissioning of existing networks for carrying, for example, water, gas or oil, are good examples where the triumvirate, that of knowledge, language and ethics, makes its presence felt. Knowledge based on experience, complemented by archives of regulatory, legislative, learned and popular texts, has to be articulated across and within groups of wellmotivated individuals and organisations involved in making decisions regarding safe design, safe maintenance, safe upgrading and safe decommissioning in part, or in whole of an in-situ network. The safetycritical systems community should consider ways in which experiential knowledge of safe operation/design and so on can be collected and used in conjunction with a given textual archive through the use of an intelligent information system. This ever burgeoning knowledge, which is simultaneously being extended and being rationalised, is articulated through an expanded vocabulary but within a restricted syntax when compared with everyday language. The knowledge, and the language in which it is articulated, are both motivated by ways in which the individuals and the organisations act. This motivation can be related to what they consider they ought to do, what obligations and duties they have. The userled SAFE-DIS project, concerned with the repair and design of urban water carrying networks, has demonstrated how the knowledge of safe design can be acquired, be formalised and be reasoned upon for autonomously generating hazard avoidance messages during the various design phases. 1 Paper to be presented at the Safety-critical Systems Symposium 1997 to be held at Brighton, England, 5-6 February 1997. The lessons learnt from this project are of relevance to the operators of equally complex energy networks, like electricity or gas networks, communication networks, and logistics networks. 1 How, what and why of a Safety-critical System The developers of safety-critical systems attempt to build systems that might behave in a prescient manner on behalf of the users of the said systems. In other words, these systems should ideally have foreknowledge and foresight of potential hazards that may jeopardise the safety of a given user environment. The assumption of prescience posits the existence of an intuitively defined knowledge of safety which has a number of facets. One facet of this knowledge is the familiarity, awareness or understanding gained through experience (or study) of making systems and environments safe. Another facet of this knowledge deals with the states or facts of knowing that something is safe to use. Yet another facet is the sum or range of what has been perceived, discovered or learned about safety. The knowledge of safety, one might claim, is collated, analysed and archived in various phases of the development of a safety-critical system. The developers of these systems use various methods and techniques such that the systems can presignify: A safety-critical system must be able to signify or intimate beforehand the existence of a hazard; once the hazard is signified or intimated, the system, in some cases, may then signify or intimate how the hazard can be avoided and how safe operation can subsequently be effected. The claim for presignification posits the existence of an intuitively defined language of safety. Such a language can be understood in a number of different senses. In one sense this language may be defined in terms of words and the methods of combining these words for identifying and avoiding hazards, and for preserving safety. A language of safety may also exist in the sense that the manner or style of expressing hazardand safety-related information is distinct from the manner or style in which other kinds of information are expressed; this may involve the use of a distinct phraseology or terminology of safety, the use of a distinctive style in which safety-related information is composed. A language of safety might also exist in the sense that this is a lingua franca of the (health and) safety community including the operatives, end-users, and strategic thinkers involved in the operation, use and conception/modification of such systems respectively. But what motivates the development of a system that is prescient and can presignify? What persuades a nuclear power plant operator to run the system safely? Why should an automotive manufacturer endeavour to use component-design software that produces a fail-safe design? How does a water company reconcile the cost of using expensive design software for designing water-carrying networks with its broader commercial objectives? Are safety-critical systems pro bono publico: public good motivated by some sense of duty or by some unarticulated moral code of conduct? A sense of obligation that motivates public good? Public good based on some idea of good/bad, right/wrong? Public good based on some notion of what things are good, right? Public good based on a conception of natural rights? The assumption that the specification and design of safety-critical systems is to a certain extent influenced by imperatives, like ought, obligation, duty, right and so on, and by judgements like the desirable, the valuable, the good , in itself suggests that there might be an ethics of safety. The interaction between the imperatives (logicals) and value-judgements (axiologicals) in avoiding hazards is exemplified by the oft-encountered "No Smoking" sign "No Smoking" is in effect a universal imperative that is effected, at a given point in time, by a singular imperative "Do Not Smoke NOW!" The singular imperative entails the value judgement: "You Ought Not To Smoke". Indeed, one might generalise here and argue that a range of prohibiting icons displayed on plant and machinery, and many of the warnings generated by safety-critical systems, reflect this entailment relationship between imperatives and value judgements; the interaction between the logical and axiological contributes to the avoidance of hazard and the preservation of safety. Safety-critical systems reflect the consensus of the interests of a number of stakeholders. One stakeholder is the vendor of a resource or service; the other stakeholder may be the end-user of the resource or service; people who are neither may also be regarded as stakeholders, e.g. regulatory bodies and public-interest lobbies. The safety of the immediate environment may be compromised by any of the stakeholders, by omission or by commission. Safety-critical systems, it appears, imply the existence of a contractarian moral theory that holds that an action, practice, law or social structure is morally permissible just in it, or principles to which it conforms, would be (or has been) agreed to by members of society under certain specified conditions. Such an approach to ethics is sympathetic to the neo-Darwinian ideals of 'market forces', 'perfect competition', 'trickledown benevolence' and so forth. This paper comprises an account of the recently completed SAFE-DIS project (Section 2) which has resulted in the development of an information system that can be used by novice engineers involved in the ‘rehabilitation’, that is repair and re-design, of urban water networks. Section 3 contains speculations about a ‘language of safety’ in terms of an idiosyncratic vocabulary and syntax used in safety-related communications. Section 4 attempts to introduce how studies in ethics can be related to the question of safety in general and to that of safety-critical systems in particular. Section 5 concludes the paper. 2 SAFE Design of Networks using Information Systems (SAFE-DIS) Project This project was a three-year (1993-1996) collaborative venture, between a university (Surrey) and a vendor of specialist software systems (Wallingford), that dealt with safety related questions regarding the safe design, cost-effective repair and the subsequent hazard-free operation of large in-situ networks. These water-carrying networks serve large conurbations, comprise hundreds if not thousands of conduits (pipes) interconnected through an equal number of nodes (including inflows, outfalls, pumps, storage locations), and changes in the design and subsequent repair, sometimes termed rehabilitation, of such networks are classed as capital projects. The SAFE-DIS project was joined by a group, the SAFE-DIS Round Table, comprising members from the private sector (three UK Water Companies), public sector (two local government-related organisations) and a UK civil engineering consultancy. Knowledge related to the safe and cost-effective rehabilitation was acquired by the SAFE-DIS project team from human experts and from specialist texts. The text corpus comprises: safety guidelines and procedures, transcripts of expert interviews, learned papers and technical notes, legal texts including the complete Water Resources Act 1991 and a 450 page book that interprets the Act. The text corpus also comprises a terminology data base. All the texts relate in one way or another to the rehabilitation of water-carrying networks. This knowledge was structured in an information system for facilitating safe and hazard free rehabilitation of a part of the network. The structured knowledge can be used to (a) help the experts examine their own knowledge, and (b) help novices to a greater or lesser degree throughout various phases of the complex rehabilitation process. The SAFE-DIS project has identified five distinct groups of software systems that may help in the five key functions that are essential for the safe rehabilitation of complex networks (see Table 1). The integration of these systems was one of the achievements of the project: Table1 Overall functionality of Safe-DIS Function Software Components Access electronic documents Full-text and hypertext management Access rules and heuristics Knowledge-base management Modelling complex network Network simulation software Sensitivity risk analyses Risk analysis software Model history and audit Report generating systems One of the important decisions of the SAFE-DIS project was to use as much off-the-shelf software as was possible without compromising the high standards that are demanded for a safety-critical system. Thus the information system has access to proprietary simulation software, risk analysis software, text analysis software, knowledge engineering tools and data base management systems. The SAFE-DIS project has also delved into the use of autonomous agents organised in a ‘society of agents’ through the use of constraint propagation. A prototype has been developed to show how autonomous agents can disseminate information about hazards to safety in a transportation network comprising a number of vehicles, drivers, freight types and transportation companies, see [Selvaratnam and Ahmad 1995]. The information system developed by the project team animates the behaviour of an experienced engineer setting a number of tasks for a lessexperienced engineer to execute. This animation is based on an industrywide rehabilitation procedure that involves over 20 specialist tasks distributed over 4 major phases (see Figure 1 below). i. Initial Planning ii. Check System Records Phase 1